Vietnam Cloud Server Rental Emergency Response Process And Responsibility Division Template

1.

Overview and Purpose

- Goal: Ensure business recovery and minimize losses within the specified SLA when cloud servers/VPS failures occur in Vietnam.
- Scope of Application: Applicable to bare metal, cloud hosts, virtual hosts, domain name resolution, CDN, and DDoS defense-related faults.
- Outputs: emergency process checklist, list of responsible persons, event logbook, and review report template. <br/>- Core indicators: first response time (<=5 minutes), temporary remission (<=30 minutes), full recovery (<=2 hours).
- Dependencies: operations and maintenance staff, network teams, CDN vendors, upstream ISPs, and customer support channels.

2.

Fault classification and preliminary assessment

- Category A (Severe): Sitewide unavailable, database crash, business cannot be externally connected, affecting 80% of > users.
- Category B (High): Single point of service exceptions, some interface failures, significant performance degradation, affecting 10%-80% of users. <br />- Category C (Normal): Log anomalies, slight delays, non-core service impact, impacting <10% of users.
- Judgment criteria: PING/HTTP availability, CPU/memory/IO load, network bandwidth uplink/downlink, error code ratio.
- Tools: monitoring alerts (Zabbix/Prometheus), traffic analysis (iftop/tcpdump), application logs (/var/log/nginx/).

3.

Emergency response procedures (standardized steps).

- Alarm Response and Classification: Within 5 minutes of receiving the alarm, the NOC confirms and classifies the person responsible for A/B/C classification, notifying the responsible person.
- Quick diagnostics: perform connectivity checks (ping, curl), port listening (ss -tunlp), and disk and memory checks (df -h, free -m).
- Mitigation measures: Initiate CDN rewinding, enable WAF or upstream cleansing during high traffic, temporarily limit rates or enable read-only mode.
- Repair and regression: Based on root cause patching (configuration adjustments, patches, scaling, database recovery), perform regression testing and close the ticket.
- Completion and review: Record the event timeline, responsible persons, and improvement plans; submit the review report and update the SOP within 7 working days.

4.

Responsibility Division and SLA (Example Template).

- NOC (Duty Engineer): Responsible for alarm confirmation, preliminary grading, and notification; SLA: First response < = 5 minutes. <br /> - Network Team: responsible for link, BGP, upstream ISP communication and cleaning rules, SLA: temporary relief < = 30 minutes. <br />- Operations Engineer: Responsible for server-level troubleshooting and repair (service restart, capacity expansion, etc.), SLA: Full recovery < = 2 hours (Category A).
- Security Team: Responsible for DDoS event analysis, WAF/ACL issuance, and coordinating vendor cleanup.
- Customer Service/Relations: Responsible for external communication and impact assessment, sending regular notifications to customers, SLA: Updates major event progress every 30 minutes.

5.

Technical solutions and command examples (including configuration examples).

- Server configuration example: Ubuntu 20.04; CPU 8 vCPU; Memory: 16GB; Disk: 200GB NVMe; Bandwidth: 1Gbps; Public IP 203.0.113.10 (example).
- DDoS protection: Operators set the cleaning threshold at 10Gbps (base), which can be upgraded to 50Gbps or connected to third-party cleaning services; CDN (example) using Cloudflare Business, cache hit rate target >=85%.
- Common command: ss -tunlp | grep 80; tail -n 200 /var/log/nginx/error.log; iptables -L -n; tcpdump -i eth0 port 80 -c 1000 -w /tmp/trace.pcap。
- Expansion and switching: Use LV expansion or cloud host online vertical expansion; Traffic switching example: Lower the domain DNS TTL to 60s, modify the A record to point to a backup node or CDN CNAME.

6.

Real-world cases and lessons learned (including configuration and SLA tables).

- Case Background: In 2025, a Vietnamese e-commerce platform (main site deployed on VN regional cloud hosts) encountered 40Gbps UDP reflected DDoS, causing public network congestion and the entire site unreachable.
- After handling: NOC confirms the alert within 2 minutes and notifies the network and security team; Within 30 minutes, carrier cleanup + Cloudflare switchover to return to origin traffic can be reduced to 2Gbps; Restore main business interfaces within 90 minutes.
- Key data: peak traffic 40Gbps, initial cleaning threshold 10Gbps, stabilized after upgrading to 30Gbps and connecting to third-party cleaning.
- Lesson: Sign upstream cleaning SLAs in advance, configure backup IPs for grayscale switching, and routinely refine DNS and CDN switching scripts.
- Improvement: Reduce the initial response time to 3 minutes, purchase a 30Gbps cleanup package and configure automatic trigger rules, and conduct regular quarterly drills.

7.

Appendix: Comparison of responsibilities and SLAs (example).

- The table below shows examples of responsibilities and SLAs for quick reference and implementation.